Discounted stochastic games poorly approximate undiscounted ones

The purpose of this note is to summarize recent results [6, 4, 5] on stochastic games by the author and his collaborators. These results have appeared or will appear in proceedings of computer science conferences. The intended reader of this note has an interest in finite stochastic games, but no particular interest in computation, that is, the computational aspects of the results obtained have been weeded out. We consider two-player zero-sum finite (but infinite duration) stochastic games G with N positions and at most m actions available to each of the two players in each position. The reward to Player I when Player I plays i and Player II plays j in position k is denoted aij . Transition probabilites are denoted p kl ij . We assume stopping probabilitites are 0, i.e., for all k, i, j we have ∑ l p kl ij = 1. To be able to state our results as simply as possible, we shall also assume throughout that for all k, l, i, j, we have aij , p kl ij ∈ {0, 1}. In particular, we assume deterministic dynamics of nature and non-negative payoffs. By G we denote the game with limiting average (undiscounted) payoffs, i.e, payoff lim inft→∞( ∑t−1 i=0 ri)/t to Player I, where ri is the reward collected by Player I at stage i. By Gλ we denote the game with payoffs discounted with a discount factor of 1− λ, i.e., with payoff λ ∑∞ i=0(1− λ)ri to Player I. By GT we denote the finite game with T stages and payoff ( ∑T−1 i=0 ri)/T to Player I. We shall be interested in the special case where G is a recursive game in the sense of Everett [3] . In a recursive game, all non-zero rewards occur at absorbing states with only one action available to each player (”terminal states”). When G is recursive, we let GT denote the finite game with payoff rT−1 to Player I (i.e., the payoff is the reward Player I collects at the last stage of play). Note that when rewards are non-negative, val(GT ) ≤ val(GT ). The seminal result of Mertens and Neyman [7] states: